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Abstract 

Ethical practices in education and research should take into consideration the publication of 
results that failed to conform to popular theories and models. This paper focuses on issues related 
to publication bias. It is possible that researchers are unaware of the threats publication bias 
poses to the validity and objectivity of research. 
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Publication Bias: A Threat to the Objective Report of Research Results 

“Unfortunately, we are not able to publish this manuscript. The manuscript is very well 
written and the study was well documented. Unfortunately, the negative results translate 
into a minimal contribution to the field. We encourage you to continue your work in this 
area and we will be glad to consider additional manuscripts that you may prepare in the 
future, (cited in Sterling, Rosenbaum, & Weinkam, 1995, p. 109). 

This letter, sent by the editor of a major environmental/toxicological journal explaining 
why a manuscript had been rejected, demonstrates some form of publication bias. Scargle 
(2000) explained that '"publication bias exists if the probability that a study reaches the 
literature, and is thus available for combined analysis, depends on the results of the study.” 
(Scargle, 2000, p. 92). Publication bias, otherwise called positive-outcome bias, is the tendency 
to publish research with a positive outcome that appears significant more frequently than 
research with a negative outcome. As the case above suggests, publication bias often takes place 
on the part of reviewers and editors who are less likely to publish negative or near neutral results, 
meaning findings that are not statistically significance. Some other sources of publication bias 
from the part of the reviewers and editors might include jealousy, revenge, and prejudice against 
certain topics, individuals, or institutions (McCutchen, 1997, as cited in Wilson, 2002). 

However, publication bias is also caused by investigators who don’t submit their negative 
outcome studies for publication, but rather hide those in their file drawers (“file-drawer effect”) 
(Dickersin, 2004; Torgerson, 2006). Research has shown that the main reasons investigators fail 
to submit their research for publication are the negative results themselves, and lack of 
motivation when the results are not interesting (Thornton & Lee, 2000). Researchers may also 
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not submit their work if they feel that the study was poorly designed and are aware of serious 
limitations (Thornton & Lee, 2000). Moreover, a consequential drive by overcommitted 
researchers to publish only their statistically significant positive studies makes them hesitate to 
submit anything less exciting for publication (Shields, 2000). Finally, another common cause is 
the perceived bias of the peer review system in favor of papers that report the detection of 
significant positive effects (Shields, 2000). Yet, the issue becomes even more serious as 
investigators do not submit their studies for publication because their new findings are either 
contrary to their earlier research or reveal that their interventions affect the subjects negatively. 
Thus, a broader definition of publication bias would suggest that it is any influence that reduces 
the amount of good science appearing in the literature (Scholey & Harrison, 2003), and it is 
damaging to the process of scientific research whether it takes place on the part of reviewers, 
journal editors, or investigators. 

There is a large body of evidence confirming that publication bias is a substantial 
problem, and it poses a serious threat to the objective report of research results. Recently, 
Torgerson (2006) conducted an extensive historical overview of problem of publication bias 
within healthcare, psychological, and educational literature. Her study disclosed that over several 
years, significant numbers of studies having non-statistically significant findings have probably 
not been published in any major journals in the fields. She concluded that for more than 40 years 
various methodological and empirical researchers have been reporting significant evidence for a 
file-drawer effect in research, yet the problem persists. How what is excluded compares to what 
is included in the literature remains an unanswered question. But what if the numerous missing 
studies systematically showed different results than those included in the literature? What are the 
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true reasons these studies remained unpublished? Are we doing scientific research, and are we 
really learning the truth? 

Researchers (e.g., loannidis, 2005; Scargle, 2000) argued that only a small number of 
studies lost in the file-drawer can produce a significant bias in psychic, medical, and social 
science research. The file-drawer effect has proven to be a major threat to the validity of 
systematic reviews and meta-analyses in nearly every field. If publication bias exists in the 
literature, researchers searching for potentially relevant studies to include in systematic reviews 
or meta-analyses will find considerably more studies with significant positive results than studies 
with significant negative results (Wilson & Lipsey, 2001). Thus, if the reasons why studies 
remain unpublished are associated with their findings, reviews or meta-analyses based only on 
published data may reach misleading conclusions. 

Apparently, once the problem exists at the lower level, it is inherited and expanded, and 
is difficult to be detected and fixed. Results from reviews and meta-analyses can be trusted only 
if all studies that have been carried out in the field of interest are fairly represented in the 
literature. Such a thing can be accomplished only if publication bias is prevented for occurring in 
the first place (Thornton & Lee, 2000). Prevention of publication bias is important both from a 
scientific perspective (sharing and consuming complete knowledge) and from the perspective of 
those who combine results from a number of studies to formulate general conclusions (reviews 
and meta-analysis) (Dickersin, 2004). 

Dickersin, Min, and Meinert (1992) argued that the “file-drawer effect” actually 
influences publication of research results more than rejection by journals. Themes associated 
with publication bias of this form include under-reporting research and hyper-claming. These 
topics are concerned with tendencies of the investigators to manipulate the truth of their research 
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in order to get published. Research on under-reporting and hyper-claming try to investigate what 
is not reported in the literature and why, as well as how objectively studies are reported. 

Rosenthal (1994) argued that one’s decision not to publish when a study has been really 
badly done, so called ''self-censoring” , is admirable and that simply starting over might be a 
better choice even though it seems waste of information. However, unfortunately, self censoring 
does not always occur for the right reasons. "Failing to report data that contradict one's earlier 
research, or one's theory, or one's values, is poor science and poor ethics. ” (Rosenthal, 1994, p. 
7). Investigators should submit and (try to) publish their negative research results because failure 
to do so contributes to publication bias. Experimental results that are disappointing or 
uninteresting, for which no significant effects could be detected, are as valuable as the positive 
and exciting ones. Of course, exciting results may worth space in more prestigious journals, but 
less thrilling findings should also be made available to those who may need to know that finding 
(Rosenthal, 1994). If negative results are not made retrievable, researchers could be wasting 
time and other resources examining empirical questions that have already been examined. 

With respect to biomedical sciences, Chalmers (2004) emphasized that underreporting 
research - or selective reporting - is scientific misconduct which can result in the continued use 
of medical treatments that are unnecessarily dangerous, ineffective, unpleasant, and costly, 
because studies that do not (re)confirm the expected benefits of those treatments remain 
unpublished. For example, in 1993, Cowley and his colleagues published a shocking article 
describing how failing to publish their own findings from a 1980 study let them repeat a serious 
mistake: 

[The 1980 study] was designed to investigate the effect of lorcainide on arrhythmias, and 
was never intended to be large enough to allow any conclusions to be reached about an 
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effect of lorcainide on survival. . . . We thought that the increased death rate was an effect 
of chance. . . and the study was therefore never published (Cowley, Skene, Stainer, & 
Hampton., 1993, p.l65). 

A similar study was again conducted by the researchers 13 years later (in 1993), where nine 
patients died among the 49 assigned to lorcainide compared with only one patient among a 
similar number assigned to placebos. Clearly, in some fields, underreporting research might 
cause a lot of damage. Although less severe, in education and social sciences, underreporting 
research may mislead competent researchers to try costly interventions associated with time- 
consuming data collection processes, attempting to replicate some desirable outcomes. 

Underreporting/ selective reporting research is not only concerned with investigator’s 
failure to report their negative results, but also with failure to report their results with sufficient 
detail to allow judgments to be made about their validity. Sales & Folkman (2000) argued that 
investigators should report their methods and analyses in a manner that allows reviewers to draw 
reasonable conclusions about the validity and generalizability of their research findings. 
'"Researchers should not report results selectively when a more comprehensive reporting would 
likely lead the reader to questions the reliability of findings. ’"{Sdle,?, & Folkman, 2000, p. 83). 

Key details and procedures such as participations rates, sampling rates, sampling biases, and data 
manipulation (e.g., deletion of outliers) should be reported. Unexpected observations, flaws, and 
limitations of the study that the investigator is aware of should not be concealed. Any additional 
data analysis that might have been undertaken by the investigator should be included in the 
manuscript, even if the findings do not support the conclusions offered (Chalmers, 1990). "In 
summary, the idea is to try to give all the information to help others to judge the value of your 
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contribution, not just the information that leads to judgment in one particular direction or 
another. ” (Feynman, 1985, as cited in Wilson, 2002). 

Perhaps the biggest evidence of under reporting research can be found in the field of 
medicine, where many publications of studies sponsored by the pharmaceutical industry ignore 
the negative results of a treatment and only report the most favorable findings. Funding sources 
can exert great pressure on scientists to suppress or minimize inconclusive results and reports of 
adverse effects. In an investigation of factors associated with the publication of research findings 
in the medical field and filed of medicine, analysts Dickersin, Min, and Meinert (1992) examined 
the fates of approved research proposals. They found that of 285 studies approved by a 
committee between 1984 and 1987 that had been completed by 1990, 138 had been published. 

Of the 138 published studies, the proportion of drug company-sponsored studies represented was 
significantly smaller than that of studies supported by government or voluntary organizations. 
The analysts concluded that data management by companies is a major reason for no publication, 
since companies often allow only favorable findings to be shared with the world. 

Maybe an opposite tendency, yet closely related to underreporting of research results, is 
overselling of research results, so called hyperclaming. Authors often exaggerate about the 
merits of their work, by reporting only a subset of the data analysis, and by avoiding a thorough 
discussion of the deficiencies of that work. An extreme example of overselling research in the 
recent years is the publication of the initial paper on Cold Fusion, by Fleischmann and Pons, in 
the Journal of Electroanalytical Chemistry in 1989 (cited in Wilson, 2002). The authors had 
gained worldwide attention by reporting that they had successfully induced a sustained nuclear 
fusion reaction at room temperature in a small jar on a laboratory tabletop. Many labs around the 
world repeatedly failed to replicate these findings within the months followed, as Fleischmann 
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and Pons had hyperclaimed about their research results. Perhaps, another example of overselling 
research would be leaving aside your initial hypothesis testing, because it did not work as you 
expected, and try multiple other hypotheses until you find something more exciting to report. 

Although, methods of detecting, correcting for, and preventing publication bias in 
literature reviews and misanalysis have been developed (loannidis, 2005), this process is not that 
simple. As discussed previously, preventing publication bias from occurring in the first place 
would be the optimal solution (Thornton & Lee, 2000). Journal editors should encourage the 
submission and publication of good quality studies regardless of the direction (positive or 
negative) of their findings. Reviewers should focus more on the quality of a study - appropriate 
theoretical framework, good design (e.g., randomized design, sufficient power), transparent 
methods and procedures, appropriate statistical analysis, valid interpretation of the findings and 
inferences - rather than on the direction of the findings (positive or negative). Good quality 
studies having null or negative results should have the same chance of being published as those 
with outcomes in the positive direction. Apparently, bad quality studies should not get published 
even if the results are promising. Moreover, reviewers should always ask for a section on the 
limitations of the study, to avoid the problem of overselling research/ hyper claming. 

Recently, a group of psychologists have made an extreme attempt to prevent publishing 
bias. In 2002, they launched a new journal dedicated to experimental studies in psychology 
which fail to reject the null hypothesis - the Journal of Articles in Support of the Null 
Hypothesis (JASNH). According to the editors, 

[Ojther journals and reviewers have exhibited a bias against articles that did not reject the 
null hypothesis. We seek to change that by offering an outlet for experiments that do not 
reach the traditional significance levels ip < .05). Thus, reducing the file drawer problem. 
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and reducing the bias in psychological literature. Without such a resource researchers 
could be wasting their time examining empirical questions that have already been 
examined. We collect these articles and provide them to the scientific community free of 
cost.” (JASNH, 2007, para.l). 

The innovative idea of having journals dedicated to the publication of null results sounds 
promising; however this method addressed the problem publication bias only to some extent. 
Giving researchers the opportunity to publish their null results in some journal, is different than 
allowing equal chances for good quality studies to be published in any journal regardless the 
direction of their findings. 

The key is to addressing publishing bias, and associated problems such as underreporting/ 
selective reporting and hyperclaming, is firstly to encourage investigators to submit good quality 
studies for publication regardless their findings; and secondly, to communicate to journal editors, 
reviewers, and investigators that null or negative results are as welcome and valuable as positive 
ones, and that focus should be on the quality of a study. From the investigators’ standpoint, 
knowing that their null or negative results are valuable to the research world might increase their 
motivation to submit their negative studies for publication (Thornton & Lee, 2000). On the other 
hand, reducing the perceived bias that reviewers favor manuscripts that report the detection of 
significant positive effects (Shields, 2000), might decrease the likelihood that investigators will 
try to mislead the reviewers (and consumers) by either underreporting or hyperclaming. 

At the expense of maybe more trouble-causing ethical issues in research, publication bias 
is sometimes given less attention in research ethics modules/courses. A quick search in the 
Internet for research ethics modules/courses showed that publication bias is not as a popular 
topic as human participants and consent, intellectual property, plagiarism, and authorship. 
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However, along with discussions on under-reporting and hyperclaming, publication bias is fairly 
well represented as a topic in ethics related journals articles. Extreme incidents of publication 
bias, used as examples in journal articles, are usually taken from the medical field; however the 
problem exists in nearly every field including education, and social sciences (Scargle, 2000; 
Torgerson, 2006). Apparently, publication bias poses serious threats to the integrity of scientific 
research. It prevents us from consuming objective, true, and unbiased research, and introduces 
much error in what we read and believe. Thus, publication bias and relevant topics should be 
addressed in research ethics modules and courses. 

Within several organizations and universities researchers (including reviewers) are often 
required to complete research ethics modules, courses, or trainings. Including topics of 
publication bias into such courses would be beneficial. Although not likely, it is possible that 
researchers are unaware of the threats publication bias poses to the validity and objectivity of 
research. Besides racing awareness, training would help investigators understand what really 
reviewers and journals editors look for, which should be good quality studies. On the other hand, 
training would help reviewers learn how to better detect problems of selective reporting and 
hyperclaming, and how to ask for important revisions and clarifications to be made, before they 
accept a manuscript for publication. Better publication practices will lead to objective and 



unbiased research, and therefore to a better world. 
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